Depth measurement quality enhancement

ABSTRACT

Described herein is a method for correcting defective depth values in depth map images. Defective values correspond to “noisy pixels” located on continuous flat surfaces and interpolated “flying pixels” located along an edge between a foreground object and a background object. The method comprising the steps of accessing a depth map of a scene which includes the foreground and background objects, detecting and identifying noisy and interpolated depth measurements within the depth map using a method, defining and applying a correction to each of the detected defective noisy and interpolated depth measurements using a specifically defined weighted correction factor. By providing the corrected defective depth values in depth map images, edges are sharpened in depth and continuous surfaces are flattened, enabling higher efficiency and robustness of further image processing.

FIELD OF THE INVENTION

The present invention relates to quality enhancement of range findingmeasurements, such as depth maps or images containing “three-dimensionalpoint cloud” data information, and is more particularly concerned withde-noising of depth measurements and the correction of problematicinterpolated depth values associated with pixels corresponding to anedge between a foreground object and a background object in an imagedthree-dimensional scene.

BACKGROUND TO THE INVENTION

Depth measurement camera systems are recent range finding measurementdevices which have become more popular due to technologies used forgesture recognition and human skeletal tracking in consumer electronicssystems and in console games.

Mainly, there are two types of environment lighting independent depthsensing or three-dimensional (3D) camera technologies that are suitablefor such applications. One type of 3D camera technology is thestructured light 3D camera, for example, provided by PrimeSense, usedfor gesture recognition in Microsoft's Kinect for Xbox 360 (known asKinect) video game console. (Microsoft, Kinect, and Kinect for Xbox 360are trademarks of the Microsoft Corporation.) A second type of 3Dsensing camera technology is the time-of-flight (ToF) camera developedand manufactured by several independent companies and which is used, forexample, in the automotive industry or for gesture recognition and humanskeletal tracking in various environments comprising human to machineinteractions, such as in video games, robotic, home automation etc.

However, regardless of the type of 3D sensing camera, an image of ascene is provided that comprises a plurality of pixels, each pixel ofthe image containing at least information relating to the distance ofthe imaged object to the camera, such information being the depth valuemeasured. Such an image embedding at least depth measurement informationis termed a “depth map”. Other types of images may also include embeddeddepth measurement information, for example, a “3D point cloud” datamatrix where images include embedded information with respect to acamera coordinate system or with respect to a virtual environmentcoordinate system. In such images, x and y correspond respectively tothe horizontal and vertical axis and the z-axis corresponds to thedepth. Transformation from a camera coordinate system to a virtualenvironment coordinate system is a matter of projections, and, suchtransformations are generally referred to as “scene calibration”.

Any application or system that makes use of images providing depthmeasurements is then dependent on measurement quality in terms ofresolution, noise, accuracy, robustness and repeatability. Inparticular, when mainly considering 3D ToF camera technologies, depthmeasurements around scene object edges are known to demonstrateconvolution and/or interpolation artefacts also termed “flying pixels”which may affect depth data in at least one-pixel radius for a singlenaturally sharp edge. Such “flying pixels” are spatial artefactsindependent from any potential motion blur at occurring in locations atedges of an object, and need to be removed and/or restored to a correctlocation in the scene which corresponds to a newly computed depth value,the newly computed depth value properly assigning the “flying pixel” toeither the foreground object or to the background object. The aim ofsuch restoration is to improve significantly subsequent object detectionconfidence and enhance depth information quality of objects within the3D scene.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide a methodand system of enhancing the quality of range finding measurements, forexample, such measurements provided in the form of a depth map or of a3D point cloud, by detecting defective pixels which may be in the formof noisy pixels values and/or in the form of interpolated “flyingpixels” values. In particular, noisy pixels values relate to the entiredepth map. However, they tend to reduce the quality of flat andcontinuous surfaces. “Flying pixels” values relate to edges of 3Dobjects represented in the depth map or in the corresponding 3D pointcloud, these edges being defined as limits in between a foregroundobject and a background object located at a different depth.

In accordance with one aspect of the present invention, there isprovided a method for depth map quality enhancement of defective pixelvalues in a three-dimensional image, the method comprising the stepsof:—

a) determining depth measurement data relating to a scene;

b) detecting defective within the depth measurement data;

c) defining a depth correction for each detected defective pixel; and

d) applying the depth correction to the depth measurement data of eachdetected defective pixel.

By using the method of the present invention, a significant enhancementof depth map or 3D point cloud image quality can be obtained as thedepth measurement noise can be specifically reduced. In particular, theinformation contained in the interpolated “flying pixels” located atedges of objects is restored and these edges are then consequentlysharpened so that to made them relevant and useful for further signalprocessing methods.

In addition, one consequence of the present invention is that user andobject detection, identification, tracking, as well as motion relateddata analysis such as gesture recognition performed on object ofinterest within a three-dimensional scene, is greatly improved as suchmethods are dependent of the depth map depth data value quality. Asanother result, extraction of images relating to user shapes and objectshapes within a scene can be performed more easily with betterreliability and accuracy.

In addition, the improved detection of objects also providessignificantly better modelling of the user and objects within the 3Dscene, in particular, human skeletal fitting and tracking is alsosignificantly improved as merging of body parts of a user with objectsor with itself is minimised and the body shape of the user can moreaccurately be modelled.

In a preferred embodiment of the present invention, step b) comprisesdetermining, for each pixel, depth related directional derivatives in atleast one direction.

Step c) preferably comprises, for each identified defective pixel, thesteps of:

c1) determining a vector in relation to the depth directionalderivatives;

c2) determining the normal to the determined vector;

c3) determining a weighting factor parameter using at least result ofone of the determined vector and the normal to the determined vectortogether with a data parameter, the value of which is in relation to themetric size in the real space represented by the pixel width; and

c4) determining a correction factor using at least one of the weightingfactor parameter and the information relating to neighbouring pixels.

In one embodiment, step c4) may further comprise using at least one ofdepth values, weighting factors, and correction factors of theneighbouring pixels. Alternatively or additionally, step c4) maycomprise using a statistical mode of the information relating toneighbouring pixels.

Advantageously, step c4) uses only valid neighbouring pixels.

Step c4) may further comprise using the depth information extracted froma regressive plane determined over the neighbouring pixels.

In one embodiment of the present invention, the defective pixels maycomprise interpolated pixel data values located at edges between aforeground object and a background object in the three-dimensionalimage. In this case, step b) may further comprise using the depthrelated directional derivatives to identify defective depth measurementsof pixels at edges when at least one depth directional derivative of apixel is greater than a predetermined threshold and if at least twoconsecutive directional derivatives have the same sign. This provides atest for the “flying pixels” as described above.

In addition to correcting for “flying pixels”, the method of the presentinvention also corrects for “noisy pixels”. In this case, step b)comprises determining defective measurements of pixels on continuoussurfaces within the three-dimensional image.

In this case, step b) further comprises using the depth relateddirectional derivatives to identify defective depth measurements ofpixels on a continuous surface when at least one depth directionalderivative of a pixel is greater than a predetermined threshold and whenanother depth directional derivative of that pixel is also greater thana predetermined threshold, the two directional derivatives havingopposite signs.

Where the defective pixels are “noisy pixels”, step c) may furthercomprise, for each identified defective pixel, the steps of:

c5) determining a vector in relation to the depth directionalderivatives data values using two orthogonal axes:

c6) determining a weighting factor parameter using at least one of aradius value of the determined vector, normal information to thedetermined vector, and real width in scene represented by the pixel; and

c7) applying a correction factor using the determined weighting factorparameter in combination with information relating to neighbouringpixels.

In one embodiment, the depth related directional derivatives aredetermined using at least two orthogonal axes. In another embodiment,the depth related directional derivatives are determined using a normalmap. In another preferred embodiment, the depth related directionalderivatives may be used for determining a normal map.

In one embodiment of the present invention, the method is used tocorrect for at least one of the “flying pixels” and “noisy pixels”, andin a preferred embodiment, the method corrects for “flying pixels” andfor “noisy pixels”.

Step a) may comprise accessing depth measurement data provided by a 3Dsensing device or camera or from a storage media in the form of a depthmap, in the form of a 3D point cloud or in any other form.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention, reference will nowbe made, by way of example only, to the accompanying drawings in which:—

FIG. 1 illustrates a schematic two-dimensional top view of athree-dimensional scene;

FIG. 2 illustrates a schematic two-dimensional front camera view of thethree-dimensional scene of FIG. 1;

FIG. 3 illustrates a schematic representation of three-dimensional datameasurement for the two-dimensional top view of FIG. 1;

FIG. 4 illustrates a schematic representation of depth value signaltogether with pixel locations;

FIG. 5 is similar to FIG. 4 but illustrates the depth value signal andpixel locations after correction in accordance with the presentinvention;

FIG. 6 illustrates a flow chart of the steps in depth map de-noising andthe edge correction method in accordance with the present invention;

FIG. 7 illustrates a 3×3 kernel centred on a “flying pixel” withadjacent pixels in two orthogonal axes; and

FIG. 8 illustrates a 3×3 kernel centred on a “flying pixel” withneighbouring pixels determined as being valid and forming part of aforeground or a background object image.

DESCRIPTION OF THE INVENTION

The present invention will be described with respect to particularembodiments and with reference to certain drawings but the invention isnot limited thereto. The drawings described are only schematic and arenon-limiting. In the drawings, the size of some of the elements may beexaggerated and not drawn on scale for illustrative purposes.

It will be understood that the terms “vertical” and “horizontal” areused herein refer to particular orientations of the Figures and theseterms are not limitations to the specific embodiments described herein.

For a typical time-of-flight (ToF) 3D camera, the process of computingdistance or depth data using TOF principles involves a combination ofoptical and electronic apparatus with analogue and digital processingunits. Typically, an amplitude modulated (AM) infrared (IR) signal issent out to the scene by the illumination system embedded in the ToFcamera device. A dedicated sensor synchronously registers the IRintensity reflected from objects in the scene as a function of phase.The signal from the sensor then is integrated over time and use is madeof phase delay computations to estimates depth value measurements foreach pixel. ToF 3D images may be limited by their resolution determinedin accordance with the sensor resolution and the field of view of thelens, their depth measurement accuracy determined in accordance with themodulation frequency of the light, the amount of light reflected by thescene and parameters associated with the imaging system, for example,the optical engine quality, the combination of optical, electronic andsignal processing apparatus that basically creates some quantisationartefacts and noise in the measurement, and more problematically, someinterpolated depth measurements leading basically to “fuzzy” edges indepth map. Such pixels in the “fuzzy” edges are also called “flyingpixels”.

The present invention is related to a method and system for resolvingthe “noisy” and “flying pixels”, collectively termed “defective pixels”,so that their depth measurement values in the depth map are corrected tocorrespond as much as possible to matter present in the scene. Themethod and system has an input comprising an image embedding depthrelated measurements, in the form of a phase map, a depth map or a 3Dpoint cloud, provided by a 3D sensing camera, a media storage device ormedia via the internet. Specific signal processing is applied to inputdepth data of the 3D image to correct for both “noisy” and interpolated“flying pixels” with respect of some specific depth gradientmeasurements, vector and geometrical constrained computations,thresholds, and more specifically weighted convolution. The resultingoutput provides a reconstructed depth map image that comprises lessnoise, and significantly less interpolated “flying pixels” around sceneobjects having edges along the z-axis or depth axis. The enhanced and/orcorrected image is then intended to be used by 3D imaging applicationsinstead of the original input 3D image provided by the 3D camera deviceor other 3D imaging system enabling better operability and efficiency ofsuch applications.

Referring initially to FIG. 1, a two-dimensional (2D) top view of ascene is shown in an x-z plane. A camera 10 has field of view defined bydotted lines 12, 14 in which a foreground object 16 and a backgroundobject 18 are present. An edge 20 of the foreground object 16 maygenerate “flying pixels” with respect to the background object 18 aswill be described with reference to FIG. 3.

FIG. 2 is a 2D front camera view of the 3D scene defined in FIG. 1 withthe foreground object 16 and the background object 18 in the x-y plane.In this view, the edge 20 is well defined and sharp between theforeground object 16 and the background object 18. Although only edge 20is shown, it will be appreciated that “flying pixels” may also bepresent at the top and bottom edges 22, 24 where the foreground object16 overlaps the background object 18. In addition, it is understood thatflat continuous surface of object 16 and 18 may also exhibit some noisypixels due to the camera sensor performances.

FIG. 3 illustrates the foreground object 32 and the background object 34in terms of pixels corresponding to the image shown in FIG. 1. As shown,two “flying pixels” 36, 38 are located on the edge between theforeground object 32 and the background object 34. Both of these “flyingpixels” may belong to either the foreground objection 32 or thebackground object 34, or only one belongs to the foreground object 32and one belongs to the background object 34. Foreground object 32 is acontinuous flat surface object, the measurements of which showing atleast one noisy pixel 37.

FIG. 4 is similar to FIG. 3 but also shows a signal 40 relating to thedepth map. As shown, in signal 40, the lower line corresponds to theforeground object indicated by pixels 42, the upper line corresponds tothe background object indicated by pixels 44, and the slope between thelower and upper lines corresponds to the “flying pixels” 46, 48. A blipin the lower line corresponds to a “noisy pixel” 43 in the foregroundobject indicated by pixels 42.

After processing in accordance with the present invention, the “noisypixel” 43 and the “flying pixels” 46, 48 are corrected as shown in FIG.5. As shown in FIG. 5, signal 50 is more defined having a straight lowerline corresponding to pixels 52 and a straight upper line correspondingto pixels 54. The “noisy pixel” shown as 43 in FIG. 4 is now correctedas shown by pixel 53 and the “flying pixels” 46, 48 have been correctlyassigned to pixels 52 corresponding to the foreground object (pixel 56)and pixels 54 corresponding to the background object (pixel 58).

In FIG. 6, a flow chart is shown that illustrates the main steps of oneembodiment of the method of the present invention. In step 60, an inputdepth map or 3D point cloud is obtained. The depth map or 3D point cloudmay be obtained directly from a 3D camera, from a media storage deviceor from the internet. In step 62, “noisy pixels” and interpolated“flying pixels”, collectively termed “defective pixels”, are detected. Ade-noising correction factor is determined for each pixel that has beendetermined as being a “noisy pixel”, and, the correction factor isapplied with respect to neighbouring pixels, step 64. Determination of acorrection factor for each pixel that has been determined as being a“defective pixel”, step 66, is then performed and applied. In step 66,the correction is carried out with respect to foreground object andbackground object depth values. In steps 64 and 66, such determinationof correction factors may include using statistical modes where thevalue assigned to the pixel being corrected are determined in accordancewith at least one of the mean, median, and/or mode values ofneighbouring pixels. The restored depth map or 3D point cloud is thenoutput (step 68). The output from step 68 can be used in any applicationwhere an accurate and reliable depth map is required or preferred.

The method of the present invention comprises two main steps, namely,detecting “noisy pixels” and interpolated “flying pixels”, andcorrecting both the detected “noisy pixels” and “flying pixels”.

For the detection of both “noisy pixels” and interpolated “flyingpixels”, a first step uses directional derivatives around a point todecide whether a pixel is “noisy” or not or “flying” or not, the pointcorresponding to the pixel being evaluated. Preferably, all of the depthimage pixels are evaluated. These directional derivatives can bemulti-directional but for ease of description only vertical andhorizontal directions are described below. It will however beappreciated that the same principles apply to other directions. Inaddition, other methods may be applied instead of using directionalderivatives.

If “P” is the pixel being evaluated in the depth map and “a” is a chosendirection in the plane, then da(P) will be the value of the derivativeat pixel “P” in direction “a”. A pixel is declared to be “flying” if theabsolute values, |da(P)| and |da+π(P)|, of the directional derivativesexceed a predefined threshold in direction “a” and if the sign of da(P)and da+π(P) are the same. A pixel is declared to be “noisy” if it has adepth value that is significantly different from all neighbouringpixels, in particular, if at least one directional derivative exceeds apredefined threshold and if at least two direction derivatives have theopposite sign. Either the “noisy pixel” test or the “flying pixel” testcan be performed for an arbitrary number of directions for each pixel.Ideally, the directions should cover a unit circle, that is, a circle ofone pixel radius. Typically, a set of directions, {a_i}, where i=1 to ncan be used with:

a _(—) i=(i−1)*π/n

Directional derivatives can be simply estimated by finite differences.In FIG. 7, a pixel 70 is the pixel “P” being evaluated with pixels 72,74, 76, 78 corresponding to the pixels to the top “T”, to the left “L”,to the right “R” and to the bottom “B” of the pixel “P”. The values ofpixels 72, 74, 76, 78 can be used to determine whether the pixel is“flying” or not and whether the pixel is “noisy” or not, in twodirections, namely, at angles of 0° and π/2 (horizontal and verticaldirections).

For these two directions, the “noisy pixel” test reduces to

(|R−P|<Th and |L−P|<Th) or (|T−P|<Th and |B−P|<Th)

and

sign(R−P)≠sign(P−L) or sign(T−P)≠sign(P−B)

The “flying pixel” test reduces to

(|R−P|>kTh and |L−P|>kTh) or (|T−P|>kTh and |B−P|>kTh)

and

sign(R−P)=sign(P−L) or sign(T−P)=sign(P−B)

where Th is the threshold value applied and k is a predeterminedweighting factor. For example, a Th value of 0.08 m can be used, but itwill be appreciated that any other suitable value can be used. As analternative to the “flying pixel” test and the “noisy pixel” test givenabove, the following can be used instead:

|L−R|>Th and |T−B|>Th

In this latter case, the value of the threshold may be larger than thatgiven in the previous test as it uses the values between two pixelssurrounding the pixel being evaluated.

“Noisy pixels” and “flying pixels” having been identified, a second stepestimates new depth values for applying individual correction to each ofthe identified “noisy pixels” or “flying pixels”. Correction can becarried out in a single pass within a single process. For a betterunderstanding, the correction steps are described as being sequentialwith respect to time.

All “flying pixels” and “noisy pixels” are first flagged as beinginvalid. In particular, when using a ToF camera, other pixels that maybe judged (that is, using thresholding) as unreliable for other reasons,for example, bad confidence, low IR illumination, can also be flagged asinvalid and their depth can also be re-estimated using the method of thepresent invention.

The principle is to use valid surrounding pixels of an invalid pixel toestimate the new depth of the invalid pixel. This is shown in FIG. 8where a pixel 80 being evaluated in relation to surround valid pixels asshown by pixel pairs 82, 84. Although pairs of pixels are shown for theevaluation, it will be understood that any suitable number of validpixels can be used for the determination.

In FIG. 8, valid pixels surrounding an invalid “flying pixel”, “P”, areindicated by “V”. However, for this estimation, only valid pixels ofeither the foreground, indicated by 82, or valid pixels of thebackground, indicated by 84, are used, and not a combination of validpixels from both the foreground and the background. Pixels 82 will havea smaller depth value than “P” and pixels 84 will have a larger depthvalue than “P”.

For a “noisy pixel”, the pixel is valid but having been identified asbeing “noisy”, the same process as described above for “flying pixels”is carried out on the “noisy pixel”.

The selection of which valid pixels to use for the estimation evaluatesa preference for the invalid point being either in the foreground objector in the background object. The preference mode can be determined, forexample, by mapping the “flying pixels” in accordance with the minimumamount of correction needed to their depth value, by setting the “flyingpixel” to the foreground object if its distance from the camera exceedsa predetermined threshold, or by setting the “flying pixel” to thebackground object if its distance to the camera is less than thepredetermined threshold. If a preference for a foreground object isused, and the set of valid foreground object pixels is not empty, thenthe new depth value of “P” will be estimated only based on those validforeground object pixels. If the set of valid foreground object pixelsis empty and the set of valid background object pixels is not empty,then the new depth value of “P” will be estimated only based on validbackground object pixels. If both sets relating to valid foreground andbackground object pixels are empty, then the pixel cannot be correctedand it remains invalid. Similarly, if a preference for a backgroundobject is used, if the set of valid background object pixels is notempty, then the new depth value of “P” will be estimated only based onthose valid background object pixels. If the set of valid backgroundobject pixels is empty and the set of valid foreground objet pixels isnot empty, then the new depth value of “P” will be estimated only basedon the set of valid foreground object pixels. If both sets relating tovalid background and foreground object pixels are empty, then the pixelcannot be corrected and it remains invalid.

The estimation of the depth value of “P” from a set of surroundingpixels (either from the foreground object or the background object) canbe made by a variety of means, including applying a weighting factor,any interpolation method using statistical determinations or using aregression plane.

In one embodiment of the present invention, a regression plane based onvalid foreground pixels is utilised. The depth value of the regressionplane at point p is assigned as the new depth value for pixel

In another embodiment, the mean depth value of valid pixels inforeground object is determined and assigned as new depth value forpixel “P”. As alternatives, the minimum, the maximum, the median or themode of the depth values of the valid foreground object pixels and/orvalid background object pixels can be used. Different estimation methodscan be used for the set of valid foreground object pixels and the set ofvalid background object pixels. For example, the maximum depth value ofthe valid pixels in the set may be used if the estimation relies onvalid foreground object pixels, and the minimum depth value of the validpixels in the set may be used if the estimation relies on validbackground pixels.

All invalid pixels whose depth values have been successfully estimatedby the method, that is, all invalid pixels that have at least one validneighbouring pixel, are flagged as being valid. The method can berepeated iteratively to allow all invalid pixels in the depth map to bereconstructed provided that at least one pixel is flagged as valid atthe beginning of the process.

However, in order to improve the reliability of “flying pixel”identification and correction, noise needs to be removed from the depthmap produced by the camera. This may be achieved by first determiningω_(i) for each pixel and then using a 3×3 weighted

_(i) kernel for each pixel. Multiple passes may be applied with orwithout re-computing the kernel

_(i) parameters.

Referring again to FIG. 8, the following equations can be used todetermine a normal angle from the depth field:—

∂x=(L−P)/2+(P−R)/2=(L−R)/2  (1)

∂y=(T−P)/2+(B−R)/2=(T−B)/2  (2)

dz=√(dx ² −dy ²)/4  (3)

dw=width of P  (4)

r=√(dz ² +dw ²)  (5)

_(i) =a cos(dz/r)  (6)

Equations (1) and (2) relate to the gradient ∂z(x) and ∂z(y) andequation (3) provides the radius in terms of the gradient. Equation (4)gives the width of the pixel “P” as stated and equations (5) and (6)provide the normal radius and the normal angle respectively.

Other methods can be used for computing, estimating or retrieving thenormal angle, for example, from the camera if available, can also beused depending on the camera, signal and platform characteristics.

For example, for computation efficiency, cpi from a depth map can bedetermined as:—

i=a cos(a tan(dz/dw))=1/√(1+(dz/dw)²)

In general, the function

i=Fw(dz) is termed a window function.

After the noise removal pass, a field gradient is calculated and thesign of second derivative d²z is used as a local disjunction parameter.Partial second degree derivative d²z is computed as the differencebetween derivative dz at both extremities of the gradient vector inprojected space.

A weighted

i 3×3 bipolar kernel is then applied in n passes, where n≧1. Thedisjunction parameter serves as group identification within the kernel;and pixel values of same sign will be averaged together while ignoringpixels with opposite sign.

This principle is improved to allow non-signed (i.e. equal to j) valuesfor the disjunction parameter such that those points can be averagedwith both signs. This improvement allows a threshold value to be appliedto the disjunction parameter, for example, using ∂²z or other data, suchas, IR power, in an effort to reduce the noise that is introduced in theoutput signal by the disjunction.

This kernel may be applied multiple times to produce desired effect.Typically current TOF signals benefit best from two-pass processing.

To accelerate this disjunction and/or avoid multi-pass processing, adisplacement map in gradient direction can be built to create localexpansion. Individual depth values can be used as well as local minimaand/or maxima values. However, noise in the output signal anddisjunction rate desired will decide which expansion method to use.

1. A method for depth map quality enhancement of defective pixel depthdata values in a three-dimensional image, the method comprising thesteps of: a) determining depth measurement data relating to a scene; b)detecting defective pixels within the depth measurement data bydetermining and using, for each pixel, depth related directionalderivatives in at least one direction; c) defining a depth correctionfor each detected defective pixel; and d) applying the depth correctionto the depth measurement data of each detected defective pixel;characterised in that the defective pixels comprise interpolated pixeldata values located at edges between a foreground object and abackground object in the three-dimensional image, and wherein step b)further comprises using the depth related directional derivatives toidentify defective depth measurements of pixels at edges when at leastone depth directional derivative of a pixel is greater than apredetermined threshold and if at least two consecutive directionalderivatives have same sign; and in that step c) comprises, for eachidentified defective pixel, the steps of: c1) determining a vector inrelation to at least one of the depth directional derivatives; c2)determining the normal to the determined vector; c3) determining aweighting factor parameter using at least one of the determined vectorand the normal to the determined vector; and c4) determining acorrection factor using at least one of the weighting factor parameterand the information relating to neighbouring pixels. 2-3. (canceled) 4.A method according to claim 1, wherein step c4) uses only data values ofvalid neighbouring pixels.
 5. A method according to claim 1, whereinstep c4) further comprises using at least one of depth values, weightingfactors, and correction factors of the neighbouring pixels.
 6. A methodaccording to claim 1, wherein step c4) comprises using a statisticalmode of the information relating to neighbouring pixels.
 7. A methodaccording to claim 1, wherein step c4) further comprises using the depthinformation extracted from a regressive plane determined over theneighbouring pixels. 8-9. (canceled)
 10. A method according to claim 1,wherein the defective pixels comprise noisy pixel data values located incontinuous surfaces of the three-dimensional image, and wherein step b)further comprises using the depth related directional derivatives toidentify defective depth measurements of pixels on a continuous surfacewhen at least one depth directional derivative of a pixel is greaterthan another predetermined threshold and when another depth directionalderivative of that pixel is also greater than another predeterminedthreshold, the two directional derivatives having opposite signs. 11.(canceled)
 12. A method according to claim 10, wherein step c) furthercomprises, for each identified defective pixel, the steps of: c5)determining a vector in relation to the depth directional derivativesdata values using two orthogonal axes; c6) determining a weightingfactor parameter using at least one of a radius value of the determinedvector, normal information to the determined vector, and real width inscene represented by the pixel; and c7) applying a correction factorusing the determined weighting factor parameter in combination withinformation relating to neighbouring pixels.
 13. A method according toclaim 1, wherein the depth related directional derivatives aredetermined using at least two orthogonal axes.
 14. A method according toclaim 5, wherein the depth related directional derivatives aredetermined using a normal map.
 15. A method according to claim 5,wherein the depth related directional derivatives are used fordetermining a normal map.
 16. A method according to claim 1, whereinstep c) further comprises, for each identified defective pixel, thesteps of: c5) determining a vector in relation to the depth relateddirectional derivatives data values using two orthogonal axes; c6)determining a weighting factor using at least one of a radius value ofthe determined vector, normal information to the determined vector, andreal width in scene represented by the pixel; and c7) applying acorrection factor using the determined weighting factor in combinationwith information relating to neighbouring pixels.
 17. A method accordingto claim 6, wherein the depth related directional derivatives aredetermined using a normal map.
 18. A method according to claim 6,wherein the depth related directional derivatives are used fordetermining a normal map.
 19. A method according to claim 7, wherein thedepth related directional derivatives are determined using a normal map.20. A method according to claim 7, wherein the depth related directionalderivatives are used for determining a normal map.
 21. A methodaccording to claim 13, wherein the depth related directional derivativesare determined using a normal map.
 22. A method according to claim 13,wherein the depth related directional derivatives are used fordetermining a normal map.
 23. A method according to claim 1, wherein thedepth related directional derivatives are determined using a normal map.24. A method according to claim 1, wherein the depth related directionalderivatives are used for determining a normal map.